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Overview 

• Constellation’s Discrete Event Simulation 

- DES? 

- Analysis 

• NASA’s Modeling & Simulation Standard 

- Analysis/Results Focused 
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CONSTELLATION 



CONSTELLATION’S 
DISCRETE EVENT SIMULATION 


Discrete Event Simulation 



• Definition: 

- Process & System Analysis, through time-based & resource 
constrained probabilistic simulation models, providing 
insight into operational system performance. 

• “Competing” types of Analysis 

- Spreadsheets 

- Scheduling Software 

- Probabilistic Risk Assessment 
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Current End-to-End CxDES Process Flow 






Post-Flight Processing 
Rofurb 
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CxDES Architec^e 
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Ariana populates 
Arena based on user 
inputs prior to run _ 
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Arena outputs 
are exported to 
Ariana post-run 
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Arena outputs are 
exported to Excel for 
post-run Individual 
mission analysis 


If running the model with 
Manifest Data and/or 
Order Lead Times, that 
data is imported from 
Excel at the start of the 
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CxDES Model Qulfiuts: 

• Process Times 


Distribution 

• Transport Times 

i 

• Cycle Times 

• Event Probabilities 1 


1 • Utilizations 

• Policies (shifting) 


— ^ • Waiting Times 


DES 

Analysis 

Cycle 


Understanding System 
Performance 

• Critical Path 

• Risk to Launch Rate 

• Margin 




Manufacturing through Launch 

Duration Comparisons 
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Manufacturing through Launch 

Duration Comparisons 



Time 
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Ares I/Orion shall be able to launch every 45 days 
Baseline {With Scrubs/Rollbacks) 



• 22% probability of 5 launches during one year 

• Average of 4.01 launches per year 
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• Average is 91.35 days between launches 



Ares I/Orion shall be able to launch every 45 days 
Baselin e (/Vo Scrubs/Rollbacks) 



Launches/yr:4 4 4 4.3 4.3 4.3 4.6 5 5.8 5.8 5.2 7.1 8.1 
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Bjsgling Integration Schedule 

6,3 Integration Schedule 

7,3 integration Schedule 

Orion time available for Integration 

Fwd Assy Manufacturing Facility Capacity Increase 

RPSF Capacity Increase 

ARF-Fwd Any Capacity Increase 

Aft Skirt Inventory Increase 

All FH€ are ready to Integrate when needed 



• Change in Integration Shift 
Schedules 
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Conclusions 



• 2 & 4 Launches per Year possible with Baseline 
Assumptions 

• ~ 90% of Cycle time is in Manufacturing & 
Assembly 

• Dependencies to 45-day launch-to-launch cycle: 

- Integration & Pad Shifting Policy 

- FHE readiness for Integration 

• Manufacturing 

• Assembly 

• Off-Line Ground Ops 

- Aft Skirt quantity (of reusable FHEs) 

• 1-time 30-day launch-to-launch cycle not possible 
using current model data 


• Input Data Refinement 
— Level 3 Projects Data 

• Automate Chart Production 

• Refine Analyses 

• Logic for minimum launch spacing 

• Adjust manufacturing start time based on 
system behavior (manage ETE Cycle Time) 

• Shelf Life of FHEs 

• Lunar SRR 
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Future Work 
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NASA’S MODELING & SIMULATION 
STANDARD (NASA-STD-7009) 


Thoughts to Discuss 

• M&S Practices 

• Reporting to Decision Makers 

• Credibility discussion 

- V&V, VV&A 

• Placarding results 
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Why a New Standard? 



• Why Aren’t Software Standards Enough? 

- Don’t cover models developed only in hardware 

• With simulations carried out as an exercise using the hardware 
models 

- M&S use is focused towards understanding a system for 
the purpose of decision making 
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Why NASA? / Why 

• Feb 1, 2003 



• Resulting Columbia 
Accident Investigation Board 
(CAIB) developed set of 
Recommendations, 
Observations, & Findings 
(R-O-Fs) 

- Directed towards the Space 
Shuttle Program 

- Some were related to Models 
& Simulations 
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Findings of Shuttle Accident 

Related to Modeling & Simulation 


Investigate 


0 


• Operating a model 
outside known 
limits 

- Conditions are 
outside known 
limits 

• Model Operator 

- Training 

- Experience 

• Assumptions 
Communicated 

- Also, Abstractions 


• Model Management 

- Maintenance 

- Support 

- Configuration 
Control 

• Data V&V (I & O) 

- Model Verified with 
Real Data 

- Model Data is 
Current 

- Sensitivity Analysis 
Performed 
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Basic Ideas 



> Documentation of M&S Activities (Sections 4.1— 4.6) 

> Credibility Assessment (Section 4.7 & Appendix B) 

> Reporting to Decision Makers (Section 4.8) 

- M&S Analysis Results 

- A statement on the uncertainty in the results 

- Credibility of M&S Results 

- Identify 

• Unfavorable outcomes 

• Violation of assumptions ^ N 

- Unfavorable Use Assessment ^ 

• Difference Between V&V & Use Assessment 

08E-SIW-076 20 
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Accreditation 


Results 


Depth of 
Development 


Depth of 
Analysis 


Depth of 
Support 


Development 

Operations 

Supporting Evidence 

Ver 

Val 

Input 

Pedigree 

Uncertainty 

Quant. 

Robustness 

Use 

History 

Model 

Mgt 

People 

Qual. 

Technical Review 



Depth of 
Review 


Use Assessment 

Fidelity 

Within 

Validated 

Domain 


V&V Foundation 
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Problem Entry 
(Syifcn) 
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© Sargent, R. G. (c. 1 980). 



Copyright © 1979 Society for Computer Simulation. 
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Verification & Validation 



Verification 

• Structure 

• Flow 

• Fidelity 

• How: 

- Comparing to 
Conceptual Model 

- Entity (Code) Tracing 

- Primitive Tests (All l’s) 

- Min/Max Value Tests 


Validation: "... determining 
the degree to which a model 
or a simulation is an 
accurate representation of 
the real world ...” 


Real 

Environment 


31 

co a 
o 


Similar 

Environment 


OK 

Siiuiloiily of 
Analogous System 
Tnusr.be carefully 
considered 

u 

Best 

Data from a previous 
flight of the same 
spacecraft in the same 
type of space environment 

■.s 
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Caution 

OK 

System & Environment 

Similarity of 

data fidelity 

Analogous Environment 

must be carefully 

must be carefully 

considered 

considered 

Rettet 



Similar Similarity Real 
System of System System 


Input Pedigree 



Input: 

• Source 

- Notional 

- Subject Matter Expert 

- Applicability to current 
problem 

• Referent Quality 
relative to current 
problem 

- Referent System 

- Referent Environment 


Input Form: 

• What’s the 

character of your 
analysis? 

- Average 


- Uniform 


- Triangular 



- Authoritative Data 


• Quantity of Source 
Data 


- Estimated PDF 



- PDF from adequate 
real-world data 
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Accuracy & Uncertainty 




Accuracy: 


True Value 


‘Modeled* Value 



High 


o £» 

d f 

Sat 


B °K 

Rest 

] At least you know 
jThat you have wide 
uncertainly 

High Confidence of a 
Narrow uncertainty 


OK 

Worst 

Low confidence of 

! Unsure of wide 
liTiOHrlaiiily 

a narrow uncertainty 
may at least have 

some maneuvering 


room 

1 Ucttci 



Wide Results Narrow 
Uncertainty 


Uncertainty: 

• Types 

• Sources 

• ‘Size’ (i.e., how big) 

• How Confident 


Epistemic 1 

Reducible 

- Subjective 

- Model Form 

- Assumptions 

- Abstractions 

- Incomplete Information 


Aleatory 

Irreducible 
(Natural) Variability 
Inherent 
Stochastic 


Uncertainty 



2 Types 

- Epistemic 

• Reducible 

• Subjective 

• Model Form 

• Lack of Knowledge 

• Incomplete Information 

- Aleatory 

• Variability 

• Irreducible 

• Inherent 

• Stochastic 

Uncertainty Occurrences 

- Parameters of the model 

- Accuracy of the model 

- Sequence of possible event 


Parametric Uncertainty 

- Aleatoric 

- Stochastic Parameters 

Model Form 

- Epistemic 

- Model Structure/Selection 

Why M&S Results may not 
be correct 

- Variability 

- Uncertainty 

- Error 

Methods 

- Representation 

- Aggregation 

- Propagation 

- Interpretation of Results 
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) 

Robustness 



Robustness of Results, i.e., 
Sensitivity of: 

• The Real World System 
(RWS) 

• The M&S 


Insensitive 
To Changes 


Sensitive 
To Changes 


Worst Situation 

M&S shows a 
Robustness not present 
in the RWS 
- Validation Issue 
- M&S not so useful 

Best Situation 

RWS is robust (Insensitive 
to Changes) 

& 

the M&S matches 
the RWS 

A. 



* 

OK Situation 

Not a Good Situation 
M&S is not robust, but 

RWS is sensitive to change 

RWS is 

& the M&S matches 

- Validation Issue 

the RWS 

- Results will be overly 
conservative 


Sensitive RWS Insensitive 
To Changes To Changes 


Use History & Management 



Use History: 

• Similarity of Uses 

- Analogous Systems 

- Exact Systems 

• Length of Time in Use 

- Just Developed 

• Just Updated 

- Long-Term Successful 
Use 


M&S Management: 

• Models & Data under 
Configuration Control 

• Models are 

- Maintained 

- Sustained 
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People Qualifications & Tech Review 


People Qualifications: 

• Education 

• Training 

• Experience 

- In M&S 

- With the Modeled (Real 
World) System 

• Use of Recommended 
Practices 


Technical Review: 


• When accomplished 

- During M&S Development 

- During M&S Operations 


Development 

Operations 

Ver 

Val 

Input 

Pedigree 

Uncertainty 

Quant. 

Robustness 


• Qualifications & 

Independence of the ‘Peer’ 
Review Group: 

- Self 

- Internal Organization 

- External 

- Non-Expert to Expert 


• Level of Formalism 

- Planning 

- Documentation 


Sample Report Formats 



BarChart 


Radar Plot 



Verification 



This briefing is for status only and does not represent complete engineering data 
analysis 


30 
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Scope of the 


M&S Standard 



• Standard covers the use of M&S affecting: 

Critical f Human Safety 1 As defined by each Program 

Decisions I • Mission Success J 


Sample Risk Matrix 


M&S 

Results 

Influence 

5: Controlling 

(G) 

(Y) 

(R) 

(R) 

4: Significant 

(G) 

(Y) 

(R) 

CR) 

3: Moderate 

(G) 

(Y) 

(Y) 

(R) 

2: Minor 

(G) 

(G) 

(Y) 

(Y) 

1: Negligible 

(G) 

(G) 

(G) 

(G) 


IV: Negligible 

III: Marginal 

II: Critical 

I: Catastrophic 

Decision Consequence 


Models / Modeling 



Modeling Aspects: 

• Incidents (events, activities) 

• Lifecycle (phases) 

• Functions 

Model Dynamics 

• Social 

• Physical 

• Environmental 

• Economic 

• Organizational 

• Infrastructure 

• Other (e.g., Engineering 
Processes 


Model Representations: 

• Conceptual 

• Mathematical 

• Dynamic 

• Programming Paradigms 

• Analytical Techniques 

Interaction Methods: 

• Live 

• Virtual 

• Constructive 

Uses / Objective: 

• Decision Support 

• Planning 

• Analysis 

• Systems Engineering 

• Training / Gaming 

• Performance Measure 

• Component / Module 
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Questions to Ask 



• Type of Analysis 

• Level of Detail 

• Type of M&S 

• Application S/W 

• Uncertainty 

• Use History 

• Config Mgt 

• V&V Domain/Range 

• Analysis Domain/ Range 
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Model Types 




Behavior Mimicking 

(Simulations) 


y = x 

Mathematical, Physical, 
or Chemical Formula 

(Algebraic Equations, ode, 
pde. Physical Formulas & 
Chemical Reaction 
Equations) 

2 //, + 0 , -> 2 H,0 


Listing & Relating 
Pieces of Information 

(Databases, Object- 
Oriented 

Hierarchy, Organizational, 
Conceptual) 

7 



Word Descriptions 

(Prose, Poetry, Req'ts 
Spec, Speech) 



Visual Form 
or Representation 

(Pictures, Graphs) 


Physical/Tangible 

(Abstract. Scaled) 
Versions 

(Model Cars, Dolls) 
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Military View of M&S 

(from an ‘Interaction Modes’ perspective) 



Real Simulated 

System System 


Live 

Virtual 

This is currently 
not defined, but 
leaves room for 
Autonomous / 
Robotic systems 
operating in a 
real environment 

Constructive 


This looks at M&S from an 
‘Interaction Mode’ perspective 
Description of categorization 
from: 

- McLean, et al. - Taxonomy 
paper - SISO 2008 

- Lee Lacey (DRC) - OneSAF 
2008 Conference 

Pink box is from conversation 
with Lee Lacey (DRC) 
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Level of Detail 



Organizational System 


Engineered System 

Engineered Component 

mm 

Engineered Part 


Physics / Chemistry 


Atomic 

Sub-Atomic 

.y 
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Network Layered Protocol Approach 


08E-SIW-076 




Layered M&S View 

(Influences in M&S Results) 



User Input 
including | 


M&S V&V 
and 

Credibility Assessment! 


Industry Standards 
and 

Broad Use 


MlS Input M/S Ou 

put 

J Model / Simulation 


t 

| Application Software 


Operating System SoftwE 

re 

1 

| Computer Hardware 



Analyzing Output 
including Post- Processing 
of Output Data 



Need for a 
Clearinghouse for 
Commercial & Open 
Source M&S 
Languages & 
Application Software 


08E-SIW-076 
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BACKUPS 


Martin’s Response 

‘Measured’ Value = M&S Result 
Comparing Values that have Uncertainty 



True Value 

Short Definitions 

Accuracy - Agreement 
between a measurement 
(M&S Result) & the True 
Value 

Uncertainty - A range of 
values likely to enclose the 
True Value 



Validation - Process of 
determining the accuracy of 
a M&S 


To know how much agreement there is 
between a measured & true value, the 
uncertainty of each must be evaluated. 
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Use Assessment 




dimensional input domain & multi-dimensional output range 


Information Reported to Decision-makers 



Additional Information 


Addition a! Information 


Section 4.7 Supplement: CAS Operational Concept 


Section 4.7 Supplement: CAS Operational Concept 


• The ban estimeie 

• Tha uncertainty statement 
a I ha CrarhtHy Assessment 

Scab remits 
a Any entrails 



a Tha bast estimate 

• Tha uncertainty alatament 

• I ha Credbsey Aatatunant 

Scab resv*s 

• Any cavmats 




Additional Information 

Section 4.7 Supplement: CAS Operational Concept 


• Tha ban estimate 

a Tha uncertainty statement 

• Tha Cradbeey Assessment Scab restiSs 

a Any caveat* 


CAUTION 


Use Assessment 

Not Performed 


This briefing is for 44 

status only and does 

nnt rpnrpQ#»nt 
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Development Progression 




CAIB 

Report 


Diaz 

Report 


Mgt 

Decision Maker 
Interviews 


Pilot 

Studies 


\ 
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SUNDAKD FORMOOILS AND SUJVLAItOM 


STANDARD TOR MODELS AND LAllONS 

UEASV 1UQM SYSTEM DO'TTffKATlOX 




Interim 


Final Submitted 

Nov 06 


Nov 07 

rot nmx wujm -tunwTWD rarcra 





2 Credibility Scales 


1 New Credibility Scale 

/ 

External NASA-wide 

Efforts Formal Review 
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Something to say about models: 




Hurricane Ivan Track Prediction Models 


46 
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Something to say about models: 



• Model Map Display from the Mid-AtlanticWX.com 
(shown on previous page) 

IMPORTANT! This map does *NOT* represent the OFFICIAL 
FORECAST TRACK! Although the "official track" may be 
included, this is not a product of the Tropical Prediction 
Center/The National Hurricane Center. 

This map is a graphic representation of computer generated 
projected tracks. This information is EXPERIMENTAL and 

subject to extreme fluctuations !); in provided fnr informntinnril 

purposes only. Do not rely orCfhisinformation! 
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Jeanne, Sept 16, 2004 - Track Prediction 



Forecast Model 

NOGAPS 

GFS (AVN) 

BAM Medium 

UKMET 
GFDL 

$ Initial position 
□ Forecast positions 
•very 24 hrs 


Hurricane Jeanne model forecasts 
NriGAPS UKMFT ran at Gpm EOT 09/15 
Gf IM Gf S ran at ?am EOT on if, 

BAMM ran at flam EOT 09 10 
c omputer models subject to larae errors 
Do not use lor planninu purposes 


SHIPS model maximum sustained winds (mph) 
uhr.GO l?hr:9u 71hr:90 

n.nr: lufl IGhr: 101 7 ? hr: 103 
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